Skip to content

🐛 Monitor Celery tasks cancellation #7514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 27 commits into from
Apr 15, 2025
Merged

🐛 Monitor Celery tasks cancellation #7514

merged 27 commits into from
Apr 15, 2025

Conversation

@giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented Apr 11, 2025 •
edited
Loading

What do these changes do?

Add an abort monitor that checks for tasks abortion to cancel the corresponding task.

CeleryWorkers don't write progress in Celerys task metadata but on separate field of a Redis hash. E.g.:

celery-task-meta:     <- keys managed by Celery (execution)
    osparc:
        42:
          <task-uuid>
          
celery-task-info:     <- keys managed by us (Hash) (info)
    osparc:
        42:
          <task-uuid>
              metadata:
              progress:

Related issue/s

How to test

Dev-ops checklist

@giancarloromeo giancarloromeo added this to the Pauwel Kwak milestone Apr 11, 2025
@giancarloromeo giancarloromeo self-assigned this Apr 11, 2025
@codecov
Copy link

codecov bot commented Apr 11, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 93.10345% with 8 lines in your changes missing coverage. Please review.

Project coverage is 87.27%. Comparing base (c5fe5bc) to head (1c15a29).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7514      +/-   ##
==========================================
- Coverage   87.44%   87.27%   -0.18%     
==========================================
  Files        1740     1370     -370     
  Lines       67389    56951   -10438     
  Branches     1143      595     -548     
==========================================
- Hits        58931    49704    -9227     
+ Misses       8138     7061    -1077     
+ Partials      320      186     -134
Flag Coverage Δ
integrationtests 65.11% <ø> (-0.03%) ⬇️
unittests 86.30% <93.10%> (-0.34%) ⬇️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library 72.65% <50.00%> (-0.02%) ⬇️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.40% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 90.02% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.52% <ø> (ø)
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.29% <ø> (ø)
datcore_adapter 98.12% <ø> (ø)
director 76.78% <ø> (ø)
director_v2 91.30% <ø> (+0.02%) ⬆️
dynamic_scheduler 97.40% <ø> (ø)
dynamic_sidecar 90.11% <ø> (ø)
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.12% <ø> (ø)
storage 87.84% <93.85%> (+<0.01%) ⬆️
webclient ∅ <ø> (∅)
webserver 85.94% <ø> (+0.06%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5fe5bc...1c15a29. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@giancarloromeo giancarloromeo marked this pull request as ready for review April 14, 2025 14:57
@giancarloromeo giancarloromeo requested a review from GitHK April 14, 2025 14:57
matusdrobuliak66
matusdrobuliak66 approved these changes Apr 14, 2025
View reviewed changes
Copy link
Collaborator

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Can you please add some description to this PR explaining a bit your changes, thanks 👍

@giancarloromeo giancarloromeo added the a:storage issue related to storage service label Apr 15, 2025
odeimaiz
odeimaiz reviewed Apr 15, 2025
View reviewed changes
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

bisgaard-itis
bisgaard-itis approved these changes Apr 15, 2025
View reviewed changes
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! Thanks a lot 🏊🏻‍♂️

@giancarloromeo giancarloromeo enabled auto-merge (squash) April 15, 2025 08:58
@giancarloromeo giancarloromeo added the 🤖-automerge marks PR as ready to be merged for Mergify label Apr 15, 2025
@sonarqubecloud
Copy link

sonarqubecloud bot commented Apr 15, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@pcrespov pcrespov requested a review from Copilot April 15, 2025 09:29
Copilot AI reviewed Apr 15, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 19 out of 19 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

services/storage/tests/unit/test_rpc_handlers_simcore_s3.py:616

  • Verify that the updated index (2) for accessing the progress value aligns with the new signature of CeleryTaskWorker.set_progress. If the call arguments change in the future, this index may need an update to avoid silent failures in progress reporting.
progress_updates = [x[0][2].actual_value for x in task_progress_spy.call_args_list]

services/storage/tests/unit/test_async_jobs.py:261

  • [nitpick] Consider replacing the magic number (60 * 10) with a named constant or adding inline documentation to clarify its purpose, ensuring future maintainers understand this value is chosen to prevent hanging tests.
payload=60 * 10,  # test hangs if not cancelled properly

pcrespov
pcrespov approved these changes Apr 15, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@pcrespov pcrespov disabled auto-merge April 15, 2025 09:45
@pcrespov pcrespov merged commit aac0abd into ITISFoundation:master Apr 15, 2025
94 checks passed
@giancarloromeo giancarloromeo deleted the monitor-celery-tasks-cancellation branch April 15, 2025 09:46
This was referenced Apr 15, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@odeimaiz odeimaiz odeimaiz left review comments

Copilot code review Copilot Copilot left review comments

@pcrespov pcrespov pcrespov approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@bisgaard-itis bisgaard-itis bisgaard-itis approved these changes

@sanderegg sanderegg Awaiting requested review from sanderegg sanderegg is a code owner

@GitHK GitHK Awaiting requested review from GitHK

Assignees

@giancarloromeo giancarloromeo

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:storage issue related to storage service

Projects

None yet

Milestone

Pauwel Kwak

Development

Successfully merging this pull request may close these issues.

Celery export_data tasks do not return when aborted

5 participants

@giancarloromeo @pcrespov @odeimaiz @matusdrobuliak66 @bisgaard-itis